As networks continue to grow in speed and complexity, the owners of these networks have an increasing need to understand and control the activity of the users of these networks. Businesses maintain large and complex internal networks that interconnect their employees and contractors and allow those individuals to access critical business information stored on servers within the business. Those same individuals also access the Internet through the business's internal network.
The need to understand and control activity of these users as they access information is driven by two business requirements. One is regulatory compliance—government and industry regulations require companies to control the behaviors of employees and contractors as they access financial information, health care records, customer's credit card data and other information. A second driver is the general need for internal security controls to prevent employees from abusing access to information. This abuse can result in theft of intellectual property, fraud, inappropriate use of critical information such as a company's unpublished financial results, and many other abuses.
Controlling the network behavior of users is made difficult by two factors. First, the number of different applications in use on the network has exploded. Many Fortune 500 companies report that there are thousands of different applications in use on their networks. Each of these applications has its own data formats and communications formats. As users examine and manipulate information on a company's servers using these applications, they generate a large number of “transactions” on the internal network. The format of these transactions and the operations performed also vary widely from application to application.
A second factor that makes control of user activity difficult is the high speed of internal transactions. The speed of internal networks can easily be several orders of magnitude faster than the edge networking speeds that connect the enterprise to the Internet. In addition, the internal network speeds continue to increase. The speed with which transactions are occurring on a company's network is exploding.
To control transactions on a company's network, the company has to translate its policies around insider/employee behavior and around regulatory compliance issues into a set of rules that control the transactions that users in various different roles are allowed to perform. These rules then need to be used to program some sort of control system that can read all of the thousands of different transactions occurring on the network and somehow apply rules to all of this traffic. The speed of the traffic and diverse nature of the transaction data makes this extremely challenging.
Rule systems typically include a “rule engine” that constantly evaluates incoming transactions against a list of rules. The list of rules is usually very long and complex, and because the transaction rate is so high, the time available to evaluate each rule gets shorter and shorter. In most rule systems, the evaluation process is to compare each transaction to every rule that is loaded. As the number of rules increases, the rule engine will run slower and slower.
The above problem is made even worse because the number of items of information in each transaction is diverse. For example, if a user accesses a file sharing system, the transaction will contain the file name, the file folder or path, the file size and many elements. These elements can have different data types; some are integers, some are fixed length strings, some are variable lengths strings, and some are binary data. This further complicates the task of the rule engine.
Shown below is how a typical rule engine might specify a rule:                IF Group(Marketing) performs READ on ANY file in path in subdirectory //users/spreadsheets on host FINANCE        THEN alert        
This rules says that people in the Marketing group may not read files stored on the FINANCE server and that are stored there in the //users/spreadsheets subdirectory. A company could easily have thousands or tens of thousands of such rules in place. And as the number of rules increases, the time to evaluate all of them for each transaction will grow. But this is the opposite of the desired result because in many cases the company wants to enforce the rules—that is, actually block the transaction before it completes. But if the rule engine takes a very long time to evaluate each transaction, that rule engine is unlikely to have finished its evaluation in time to stop the transaction. And even if the rule engine is fast enough to initially stop the transaction, there is no guarantee that it will be able to do so in the future as new rules are added because these new rules will slow down the rule engine.
Accordingly, a need exists in the art for a rule engine having the ability to evaluate a wide variety of different transaction types each containing a wide variety of elements that have many different data types. A further need exists for such a solution that has the ability to compare a transaction against a very large number of rules in a very short period of time (less than a millisecond), and have that evaluation time remain fixed regardless of the number of rules loaded.